Papers with structured pruning
Iterative Structured Pruning for Large Language Models with Multi-Domain Calibration (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing models with unstructured pruning often yield irregular sparsity patterns that necessitate specialized hardware or software support. |
| Approach: | They propose a structured pruning framework that eliminates entire architectural components and maintains compatibility with standard hardware accelerators. |
| Outcome: | The proposed model pruning framework achieves significant compression with minimal performance degradation on multiple models across diverse downstream tasks. |
Structured Pruning for Large Language Models Using Coupled Components Elimination and Minor Fine-tuning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated powerful capabilities in natural language processing, yet their vast number of parameters poses challenges for deployment and inference efficiency. |
| Approach: | They propose a structured pruning algorithm that derives the importance of different components based on intermediate data dependencies and removes coupled components across different layers simultaneously. |
| Outcome: | The proposed algorithm reduces model size and accelerates inference without specialized operators and libraries, while maintaining its utility as versatile problem solvers. |
LLMs on a Budget? Say HOLA (2025.emnlp-industry)
Copied to clipboard
Zohaib Hasan Siddiqui, Jiechao Gao, Ebad Shabbir, Mohammad Anas Azeez, Rafiq Ali, Gautam Siddharth Kashyap, Usman Naseem
| Challenge: | Current solutions such as quantization, pruning, and Retrieval-Augmented Generation (RAG) offer only partial optimizations and often sacrifice accuracy, speed, or generality. |
| Approach: | They propose an end-to-end optimization framework for efficient LLM deployment . it leverages Hierarchical Speculative Decoding (HSD) for faster inference without quality loss. |
| Outcome: | HOLA delivers +17.6% EMA on GSM8K, +10.5% MCA on ARC, and reduced latency and memory on edge devices like Jetson Nano. |
Less Is More: Domain Adaptation with Lottery Ticket for Reading Comprehension (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain adaptation paradigms for reading comprehension require large amounts of annotation data to achieve the desired task performance. |
| Approach: | They propose a few-shot domain adaptation paradigm for reading comprehension . they introduce self-attention attribution to weigh parameters and refine the lottery subnetwork . |
| Outcome: | The proposed model outperforms the full model fine-tuning adaptation on four out of five domains with a small amount of data available for adaptation. |
Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems (2025.emnlp-industry)
Copied to clipboard
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications. |
| Approach: | They propose two techniques for training and deploying small language models that deliver high performance for a variety of industry use cases. |
| Outcome: | The proposed techniques retain much of the quality of larger models while reducing training/serving costs and latency. |
Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency (2022.acl-long)
Copied to clipboard
| Challenge: | Structured pruning has been extensively studied on monolingual pre-trained models . but little attention has been paid to evaluating the effectiveness of structured pruning on multilingual models. |
| Approach: | They investigate settings, algorithms, and efficiency of structured pruning on multilingual models . authors propose a simple approach that allows training the model once and adapting to different model sizes at inference . |
| Outcome: | The proposed approach allows training the model once and adapting to different model sizes at inference. |
QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Structured pruning can reduce model size but results in significant accuracy degradation . quantization and pruning increase the difficulty of fine-tuning, requiring a more refined quantization scheme. |
| Approach: | They propose a structured pruning framework followed by a layer-wise mixed-precision quantization scheme to reduce model memory consumption during fine-tuning and inference. |
| Outcome: | Experiments on benchmark datasets show that QPruner outperforms existing methods in memory savings while maintaining or improving model performance. |
When BERT Plays the Lottery, All Tickets Are Winning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Large Transformer-based models are reduced to a smaller number of self-attention heads and layers. |
| Approach: | They propose to prune BERT self-attention heads and layers to find subnetworks with comparable performance . they also extend this technique to multi-layer perceptrons to find out if they are unstable . |
| Outcome: | The proposed models are able to achieve 90% of full model performance with structured pruning and similar-sized subnetworks sampled from the rest of the model perform worse. |
OCP: Outlier-Centric Probing for Dynamic Structured Pruning of LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing structured pruning methods fail to identify outlier-triggering tokens and uniform layer-wise sparsity misaligns with heterogeneous outlier distributions. |
| Approach: | They propose a framework that prioritizes capturing outlier-triggering tokens rather than reconstructing full hidden distributions. |
| Outcome: | Experiments on LLaMA2, LLama3 and OPT show that the proposed framework outperforms state-of-the-art methods and achieves 25% perplexity reduction at 1.6 speedup. |
Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in hardware, modeling, and optimization for deep neural networks have led to improvements in memory and inference efficiency. |
| Approach: | They propose to combine sharpness-aware minimization with various model compression methods to improve model compressibility. |
| Outcome: | Empirically, optimizing for flatter minima leads to greater compressibility of parameters compared to vanilla Adam when fine-tuning BERT models, with little to no loss in accuracy on the GLUE text classification and SQuAD question answering benchmarks. |
NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Structured pruning methods have proven effective in reducing the model size and accelerating inference speed in various network architectures. |
| Approach: | They propose a framework that narrows the encoder and shortens the decoder networks of encoder-decoder models. |
| Outcome: | The proposed framework reduces the number of decoder layers and improves generation quality. |
Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation (2025.coling-main)
Copied to clipboard
| Challenge: | Structured pruning and knowledge distillation are often not efficient and require a fixed architecture, limiting flexibility. |
| Approach: | They propose a method which integrates knowledge distillation and structured pruning by replacing transformer blocks with smaller, efficient versions during training. |
| Outcome: | The proposed method outperforms L1 pruning and maintains four-fifths of performance on language modeling and commonsense reasoning tasks. |
GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models are expensive to serve because dense FFN blocks, multi-head attention, and KV caches dominate memory. |
| Approach: | They propose a global budgeted structured pruning framework that prunes FFN channels and attention KV head groups under a single global parameter budget. |
| Outcome: | The proposed model removes 50% of parameters and achieves 12.18 perplexity on WikiText-2 while maintaining competitive average zero-shot accuracy on five downstream benchmarks. |
CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information (2025.coling-main)
Copied to clipboard
Yuxin Wang, MingHua Ma, Zekun Wang, Jingchang Chen, Shan Liping, Qing Yang, Dongliang Xu, Ming Liu, Bing Qin
| Challenge: | Existing LLM pruning works focus on unstructured pruning, which typically requires special hardware support for a practical speed-up. |
| Approach: | They propose a network pruning framework that leverages both coarse and fine-grained activation information as an importance criterion to guide pruning. |
| Outcome: | The proposed framework outperforms existing pruning methods on diverse models across sparsity budgets. |
PruMUX: Augmenting Data Multiplexing with Model Compression (2023.findings-acl)
Copied to clipboard
| Challenge: | Prior work has investigated methods like model pruning, knowledge distillation, and data multiplexing to increase model throughput without sacrificing accuracy. |
| Approach: | They propose to combine structured pruning and data multiplexing methods to increase model throughput without sacrificing accuracy. |
| Outcome: | The proposed method achieves 7.5-29.5X throughput improvement over a BERT-base model with accuracy threshold from 80% to 74%. |